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The  Four-Element  Framework: 
An  Integrated  Test  and  Evaluation 

Strategy 
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Background 


DoD 

5000.1 


JCIDS 


r 


DOD  5000.1  -  “The  primary  objective  of  Defense  acquisition  is  to 
acquire  quality  products  that  satisfy  user  needs  with 
measurable  improvements  to  mission  capability...” 

Joint  Capabilities  Integration  and  Development  System 

•  War  Fighting  Capability  Gaps 

•  Material/Non-material  Solutions 


Materiel  System 
Performance  Attributes 
Key  Performance  Parameters 


Capabilities 

Development 

Document 

Capabilities 

Production 

Document 

_ F 

DoD  Architecture  Framework  Products 


Operational  View  -  mission  tasks,  aetivities,  operational  elements 
and  information  required  to  accomplish  warfighting  mission. 

System  View  -  system  elements  and  capabilities  necessary  to 
support  warfighting  functions. 

Technical  View  -  set  of  rules  and  standards  to  ensure  that  a 
system  satisfies  a  set  of  operational  requirements. 

All  View  -  overarching  architecture  that  supports  the  OV,  SV  and 
TV. 


Army 


AND  Evaluation  Command 


Introduction 


T&E  Process  Paradigms: 


Traditional; 


Proposed 


Determine 

Mission 

Needs 


Evaluate  Mission 
Performance 


Capability 
Gaps 


Develop 

System 

Requirements 


Evaluate  System 
Performance  & 
Suitability 


KPPs 
ttributes 


KPPs 
ttributes, 


Test  System 
Performance  & 
Suitability 


Completes  the  Feedback  Loop  to  Mission  Needs 


\ 

/ 
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Overview 

The  Four  Elements 


Mission 

Perspective 


T&E 

Perspective 


Purpose 

(What) 

MISSION  ELEMENT 

Mission  Tasks  and  Sub-tasks 

Purpose 

(What) 

Means 

(How) 

SYSTEM  ELEMENT 

System  and  Sub-system  Functions 

EVALUATION  ELEMENT 

Mission  Ability  and  System  Capability  Measures 

Means 

(How) 

TEST  ELEMENT 

Data  Products  and  Data  Sources 
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Overview 

Elements,  Interfaces  and  Traces 


ELEMENTS  AND  TRACES 


TEST 

ELEMENT 


PLANNING  TRACE 
EVALUATION  TRACE 

ELEMENT  TO  ELEMENT 
INTERFACE 


Elements 

•  Mission,  System,  Evaluation, 
and  Test 
Interfaces 

•  Mission  to  System 

•  Mission  to  Evaluation 

•  System  to  Evaluation 

•  Evaluation  to  Test 
Traces 

Planning  =  Mission  to  Test 
Evaluation  =  Test  to  Mission 
Two  Types: 

•  Type  1  links  Mission,  System,  Evaluation  and  Test  Elements. 

♦  Plans  and  evaluates  mission  task  ability  through  system  funetion  eapability. 

•  Type  2  links  Mission,  Evaluation  and  Test  Elements. 

♦  Plans  and  evaluates  mission  task  ability  direetly. 


SYSTEM 

ELEMENT 

. . . 

EVALUATION 

ELEMENT 

< 


MISSION 

ELEMENT 


Army 


AND  Evaluation  Command 


Purpose 


Element/Interface  Development 

Mission  Element 


•  To  describe  unit  mission  and  tasks. 

—  A  task  is  defined  as  a  discrete  action  that 
the  unit  (system  and  its  operators)  must 
perform  in  order  to  accomplish  its  mission. 


Components 

•  Critical  Operational  Objective:  Mission 
based  -  “How  capable  is  the  (unit  and 
system)  in  supporting  (mission  statement) 
in  an  operational  environment.” 


•  Task  Levels:  Orderly  breakdown  of  the 
mission  into  tasks  and  sub-tasks. 


•  Alternate  Mission  Tasks:  Optional 
mission  tasks  used  to  accomplish  part(s) 
of  the  mission.  Alternate  task  options  define  different  “mission  threads.” 


COI:  How  capable  is  the  (unit  and  system)  in  supporting 
(mission  statement)  in  an  operational  environment? 


Level  0  (Mission) 


[OA] 

Mission 


Level  1  (Task) 


[0.1] 

[0.2] 

_ ^ 

[O.n] 

Task1 

Task? 

- 

Task  n 

[0.2.1] 
Sub-task  of 
Task? 


[0.2.2] 

2^^  Sub-task  of 
Task? 


Level  2  (Task) 

[0.2.n] 

- H  Sub-task  of 

Task? 


[0.2.2.2A] 

[0.2.2.3A] 

!-► 

2'^^  Sub-task  of 

— ► 

3''^  Sub-task  of  - 

Task  2.2 

Task  2.2 

Level  3  (Task) 


[0.2.2.2B] 
Alternate  2^^  Sub¬ 
task  of  Task  ?.? 


[0.?.?.n] 
Sub-task  of 
Task  2.2 
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Element/Interface  Development 
Mission  Element  -  Example 

Development  Keys 

•  Temporal  Format. 

♦  Temporal  format  provides  a  block  diagram  of 
mission  to  mission  tasks  in  order  of  their 
occurrence. 

♦  Supports  development  of  mission  threads. 

•  Lowest  Level  of  Mission  Tasks. 

♦  Lowest  level  mission  tasks  must  be  measurable. 

♦  Evaluated  directly  or  indirectly  via 
evaluation  of  system  function  capability. 


•  Support  Documents. 

♦  Mission  Need  Statement,  Initial  Capabilities 
Document,  Operational  and  Organizational 
Plan,  Universal  Task  Lists,  Capabilities 
Development/Production  Documents 
(CDD/CPD). 

♦  Integrated  architecture  products  in  CDD/CPD 
uniquely  support  mission  element. 

=  OV-I:  Who,  How,  Where,  When,  Why  of  the  system  and  its  mission. 

=  OV-5:  Operational  activities  (mission  tasks). 

=  OV-6c:  Association  of  capabilities  with  sequences  of  operational  activities  (mission  tasks).  9 


MISSION  ELEMENT  -  Unmanned  Aerial  System  (Example) 
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Element/Interface  Development 

System  Element 

Purpose 

•  To  describe  the  system  and  the  system 
functions  and  characteristics. 

Components 

•  System  Items:  Makeup  of  the  system 
and  sub-systems. 


•  System  Functions:  Description  of  the 
function  an  item  must  perform  in  support 
of  the  mission. 


•  System  Characteristics:  Description  of  a 
particular  quality  of  the  system  that  effects 
whether  the  item  can  perform  a  function. 

•  System  Level:  Level  of  systems,  sub¬ 
system,  and  components  from  the  system- 
of-systems  perspective. 


1st  System 


1.C1 

Characteristic  of 
the  System 


1.F1 

Function  of  the 
Integrated  System 


2"^  System 


2.C1 

Characteristic  of 
the  System 


System  of  Systems 


3*^^  System 


3.C1 

Characteristic  of 
the  System 


3.C2 

Characteristic  of 
the  System 


1.1 

1^^  Sub-system  of 
System  1 


1.1. FI 

Function  of  Sub- 
system  1.1 


1.2 

2*^^  Sub-system  of 
System  1 

1.2. C1 

Characteristic  of 
the  Sub-System 

1.2. F1 

Function  of  Sub¬ 
system  1.2 


2.2 

Sub-system  of 
System  2 


2.2.C1 

Characteristic  of 
the  Sub-System 


2.2.C2 

2^^  Characteristic 
of  the  Sub-System 


2.2.F1 

Function  of  Sub- 
system  2.2 


Level  1 

(System/Sub-system) 


3.1 

Sub-system  of 
System  3 

3.1. FI 

1®^  Function  of 
Sub-system  3.1 


3.1. F2 

2^^  Function  of 
Sub-system  3.1 


2.2.1 

Sub-system  of 
System  2.2 


2.1.1. FI 

Function  of  Sub¬ 
system  2.2.1 


Level  2 

(System/Sub-system) 
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Element/Interface  Development 
System  Element  -  Example 

Development  Keys 

•  Item  to  Function  Link. 

♦  Objective  is  to  define  the  system  functions. 

♦  System  item  is  the  sub-system  responsible 
for  providing  the  function. 

•  System-of-Systems. 

♦  Include  systems  that  are  not  part  of  the 
system  being  developed  and  evaluated  if 
they  are  required  to  support  the  mission. 

•  Lowest  Level  of  System  Function. 

♦  Should  be  associated  with  the  accomplishment 
of  a  mission  task. 

♦  Measurable  by  T&E. 

•  Risk  Areas 

♦  Items  and  functions  can  be  based  on  a  specific 
area  of  developmental  risk. 

•  Support  Documents. 

♦  System  Work  Breakdown  Structure 

♦  Integrated  architecture  products  in 
CDD/CPD  uniquely  support  mission  element. 

=  SV-1 :  Systems  required  to  support  the  mission  and  the  interfaces  between  them. 

=  SV-4:  System  functions  required  to  support  the  operational  activities  (mission  tasks). 
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Element/Interface  Development 
Mission  to  System  Interface 

Purpose 

•  To  describe  how  the  mission  tasks 
relate  to  the  system  functions. 

Components 

•  Mission  Tasks:  Taken  from  the 
mission  element. 

•  System  Functions  and 
Characteristics:  Taken  from  the 
system  element. 

•  Input  Rule:  Description  of  how 
the  system  items  relate  to  the  mission 
task.  Uses  logical  input  rules,  such 

as  AND  and  OR  to  describe  links  to  more  than  one  system  or  function. 


[0.1] 

Task  1 

[0.2.1] 

1®t  Sub-task  of 

1  Task  2 

[0.2. 2.1] 

1®*  Sub-task  of 
Task  2.2 

[O.n] 

Task  n 

System  1 

1.C1 

Characteristic  of 
System  1 

Input  Rule 
(AND/OR) 

Input  Rule 
(AND/OR) 

Input  Rule 
(AND/OR) 

1.1. F1 

Function  of  Sub- 
system  1 . 1 

Input  Rule 
(AND/OR) 

Input  Rule 
(AND/OR) 

Conditions 

Conditions 

1.2. FI 

Function  of  Sub¬ 
system  1 .2 

Input  Rule 
(AND/OR) 

Conditions 

System  2 

2. 2. Cl 

Characteristic  of  Sub¬ 
system  2.2 

Input  Rule 
(AND/OR) 

Input  Rule 
(AND/OR) 

2.2. FI 

Function  of  Sub¬ 
system  2.2 

Input  Rule 
(AND/OR) 

Conditions 

2.2. FI 

Function  of  Sub¬ 
system  2.2 

Input  Rule 
(AND/OR) 

Conditions 

System  3 

3.C1 

Characteristic  of 
System  3 

Input  Rule 
(AND/OR) 

3.1. FI 

1®^  Function  of  Sub¬ 
system  3.1 

Input  Rule 
(AND/OR) 

Conditions 

•  Conditions:  Description  of  the  physical,  military,  and  civil  variations  that  effect 
performance  of  a  task.  For  example;  weather  conditions,  countermeasures,  urban 
environment,  etc. 


Army  Xest  and  Evaluation  Command 


Element/Interface  Development 
Mission  to  System  -  Example 


Development  Keys 

•  Input  Rule. 

♦  Link  every  function  required  to  support 
the  mission  task. 

♦  Link  alternate  system  functions  that 
support  the  mission  task. 

♦  Top  row  for  every  system  defines  if  the  system 
supports  the  mission  task  with  a  function. 

(Used  later  to  link  system  suitability  to  the  task.) 

♦  Linkages  are  important  since  they  will  be 
used  to  evaluate  mission  tasks  based 

on  the  evaluation  of  system  functions/suitability. 

•  Conditions. 

♦  Consider  the  conditions  based  on  the  ability  to 
support  the  mission  task,  but. . . 

♦  The  specific  function  may  drive  the  choice 

of  applicable  conditions.  For  example;  terrain 
may  effect  the  communication  functions  of 
line-of-sight  systems  but  not  effect  satellite 
systems. 

•  Support  Documents. 


MISSION  TO  SYSTEM  LINKS  -  Unmanned  Aerial  System  (Example) 


♦  Initial  Capabilities  Document  and  System  Threat  Assessment  Report  to  determine  conditions. 


♦  Factors  of  METT-TC  to  determine  conditions. 


♦  Integrated  architecture  products  in  CDD/CPD  uniquely  support  mission  element. 

=  SV-5:  Maps  operational  activities  (mission  tasks)  from  the  OV-5  to  the  system  functions  from  the  SV-4.  13 
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Element/Interface  Development 
Evaluation  Element 


Purpose 

•  To  describe  the  evaluation  measures 
and  how  they  relate  to  mission  tasks, 
system  functions,  and  system  suitability. 


COI:  Does  the  (system)  perform  (system  capability)? 


1.C1.S1.P1 
MOP  for  MOS 
1.C1.S1 


1.1.F1.E1.P1 

MOPfor  MOE 
1.1.F1.E1 


1.1.F1.E2.P1 

MOPfor  MOE 
1.1.F1.E2 


Standard  for 
MOE 
1.1.F1.E2 


1.1.F1.E2 
2nd  mOE  for 
Function 
1.1. F1 


1.2. FI 

Function  of  Sub¬ 
system  1.2 


Input  Rule 
(AND/OR) 


Conditions 


2.2.C1.S1.P1 
MOPfor  MOS 
2.2.C1.S1 


Standard  for 
MOS 
2.2. Cl. SI 


2.2.C1.S1 
MOS  for  Sub¬ 
system  2.2 


2.2.C1 

Characteristic  of 
Sub-system  2.2 


Input  Rule 
(AND/OR) 


2.2. FI. El. PI 
MOPfor  MOE 
2.2.F1.E1 


Standard  for 
MOE 
2.2.F1.E1 


2.2. FI. El 
MOE  for 
Function 
2.2.F1 


2.2. FI 

Function  of  Sub¬ 
system  2.2 


Input  Rule 
(AND/OR) 


2.2.F2.E1.P1 
1st  MOPfor  MOE 
2.2. FI. El 


2.2.F2.E1.P2 
2nd  MOP  for  MOE 
2.2. FI. El 


Standard  for 
MOE 
2.2.F2.E1 


2.2.F2.E1 
MOE  for 
Function 
2.2.F2 


2.2. F2 

Function  of  Sub¬ 
system  2.2 


Components 

•  Conditions:  Conditions  are  assigned  to  tasks 
that  are  linked  directly  to  a  MOE  in  the 
evaluation  element. 

•  Measure  of  Effectiveness  (MOE):  Parameter 
used  to  evaluate  the  system  function  or 
mission  task. 

•  Measure  of  Suitability  (MOS):  Parameter 
used  to  evaluate  the  suitability  of  a  system. 

•  Standard:  Acceptable  performance  of  the 
system  function  or  mission  task  in  terms  of  the 
MOE  or  MOS. 

•  System-focused  COI:  COI  focused  on 
system  or  sub-system  performance.  Typically 
stated,  “Does  the  (system)  perform  (a  specific  required  capability)?” 

•  Link  to  System-focused  COI:  Column  in  the  evaluation  element  that  identifies  which  MOE/Ss  are 
used  to  evaluate  the  system-focused  COI. 

•  Measure  of  Performance  (MOP):  Quantitative  or  qualitative  measure  of  system  performance  under 
specified  conditions. 


0.2.1. El. PI 

MOPfor  MOE 
0.2.1. El 


0.2.2.1.E1.P1 

MOPfor  MOE 
0.2.2.1.E1 


Standard  for 
MOS 
1.C1.S1 


Standard  for 
MOE 
1.1. FI. El 


Standard  for 
MOE 
0.2.1. El 


Standard  for 
MOE 
0.2.2.1.E1 


1.C1.S1 
MOS  for 
System  1 


1.1. FI. El 
l^t  MOEfor 
Function 
1.1. FI 


1.C1 

Characteristic  of 
System  1 


1.1. FI 

Function  of  Sub¬ 
system  1.1 


0.2.1. El 

MOEfor  Task  0.2.1 


0.2.2.1.E1 

MOE  for  Task  0.2.2.1 


[0.1] 
Task  1 


Input  Rule 
(AND/OR) 


Input  Rule 
(AND/OR) 


Conditions 


[0.2.1] 

1®*  Sub-task 
of  Task  2 


Input  Rule 
(AND/OR) 


[0.2.2. 1] 
1®*  Sub-task 
of  Task  2.2 


Input  Rule 
(AND/OR) 


Input  Rule 
(AND/OR) 


Input  Rule 
(AND/OR) 


Input  Rule 
(AND/OR) 


Conditions 
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EVALUATION  ELEMENT  -  Unmanned  Aerial  System  (Example) 


COI:  Does  the  missile  guide,  fly  to 
and  impact  the  target  in  its 
intended  operating  environment? 


Measures  of  Performance 


1.C1.S1.P1  MTBMEF 


1.1. F1.E1.P1  %  of  accurate 
sent  messages. 

1.1. F1.E2.P1  %  of  complete 
messages. 


1.1.F1.E2.P3  Time  of  drop  out. 


KEY 


EVALUATION 


CONDITIONS 


1.C1.S1  Mission  Rel 


1.C1  Reliability 


Element/Interface  Development 
Evaluation  Element  -  Example 

Development  Keys 

•  Mission  and  System  Elements. 

♦  All  system  functions  must  have  at 
least  one  MOE. 

♦  Mission  tasks  linked  directly  to  a 
MOE  usually  indicate  a  need  for 
evaluation  during  OT&E. 

•  MOEs,  MOSs  and  MOPs. 

♦  System  functions  and  mission  tasks 
may  have  more  than  one  MOE. 

♦  MOEs  may  have  more  than  one  MOP. 

♦  Both  systems  and  sub-systems  may 
have  one  or  more  MOSs. 

♦  “Dry  run”  evaluation  from  MOP 
to  mission  task  to  ensure  evaluation 
is  sound. 

•  Standards. 


COI:  How  capable  is  the  UAS  equipped  unit  in  supporting  the  Commander’s 
RSTA  and  Armed  RSTA  requirements  in  an  operational  environment? 


<  5.0%,  <  30 
seconds 

(Attribute) 


1.1.F1.E1  Data 
Accuracy 


1.1.F1.E2  Drop  Out 
Rate 


Ground 

Control 

Station 


1.1.F1  Communicate 
with  AV 


1 .3.F1  .E1  .P1  %  via  direct  route. 


1.3.F1.E1.P2  %  via  waypoints. 


Military 

Judgment 


1.3.F1.E1%of 
Successful  Course 
Changes 


1.3.F1 

Navigate  AV 


2.C1.S1.P1  #MTTR 

<  4  hrs  (KPP) 

2.C1.S1  Reparability 

2.0 

UAV 

2. Cl  Maintainability 

2.2.F1.E1.P1  Stationary  Targets 

>  xx.x  %  at  XX 
km  (KPP) 

2.2.F1.E1  %of 

Targets  Detected 

2.2.F1 

Detect  Target 

2.3.F1.E1.P1  Difference 
between  estimated  and  actual 
time  of  arrival. 

<10  sec  from 
estimated  time 
of  arrival  (AA) 

2.3.F1.E1  Waypoint 
Arrival  On-Time  % 

2.3.F1 

Fly  to  Waypoint 

2.3.1.F1.E1.P1  %  of  Successful 
T/0 

Must  Control 
YES/NO  (AA) 

2.3.1. FI. El  %of 
Successful  T/0 

2. 3.1. FI  Control 

Takeoff  and  Landing 

2.3.1. F1.E2.P1  %  of  Successful 
Landings 

Must  Control 
YES/NO  (AA) 

2.3.1.F1.E2%of 
Successful  Landings 

3.C1.S1.P1  MTBSF 

X 

>50  hrs  (KPP) 

3.C1.S1  In-flight  Rel 

3.0 

Weapon 

3. Cl  Reliability 

3.1. FI 

Guide  and  Hit  Target 

3.1. FI. El. PI  %  targets  hit. 

X 

Performance 
similar  to  AGM- 

XXX 

3.1.F1.E1 

Probability  of  Single 
Shot  Hit 

0.1.E1.P1  Time  to  plan. 


0.1.E1.P2  %  successful  loads. 


0.2.3.3.E1.P1  %  operational 
targets  detected. 


Military 

Judgment 


0.1. E1  %  of  Successful  Mission  Planning  Sessions 


0.2.3.3.E1  %  of  Targets  Detected 


♦  Assign  a  standard  to  each  MOE  to  assist  in  resolution  of  the  MOE.  Typically  four  types  of  standards: 

=  Direct  Measurement:  Compare  demonstrated  performance  to  standard.  For  example;  maximum  range. 

=  Pass/Fail:  Demonstration  of  a  particular  feature.  For  example;  required  number  of  hard  points. 

=  Comparison:  Compare  performance  of  two  systems.  For  example;  “performance  equal  to  or  greater  than. . 
s  Military  Judgment:  No  specific  standard.  Military  utility  will  be  determined  after  the  evaluation.  15 
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Element/Interface  Development 

Test  Element 


Purpose 

•  To  describe  the  data  products,  the 
sources  of  the  data  products,  and 
how  they  relate  to  the  evaluation 
element’s  MOPs. 

Components 

•  Link  to  MOPs:  Description  of 
which  data  products  support  which 
MOPs. 

•  Data  Products:  Specific  data  packet 
obtained  though  a  data  source 
satisfying  a  MOP  data  requirement. 


Operational 
Test  Event  #2 

Operational 
Test  Event  #1 

Modeling 

and 

Simulation 

Developmental 

Test 

Contractor 

Test 

DATA 

SOURCE 

MOPs 

Data  Product  #1 

Data  Product  #2 

Data  Product  #1 

Data  Product  #2 

Data  Product  #3 

Data  Product 

Data  Product  #1 

Data  Product  #2 

Data  Product  #3 

Data  Product  #1 

Data  Product  #2 

X 

X 

MOP 

1.C1.S1.P1 

X 

X 

X 

X 

X 

MOP 

1.1.F1.E1.P1 

MOP 

1.2.F1.E1.P1 

MOP 

2.2.C1.S1.P1 

X 

MOP 

2.2.F1.E1.P2 

X 

X 

MOP 

0.2.1.E1.P1 

X 

MOP 

O.n.ElPI 

•  Data  Sources:  The  specific  source  of  a  data  product. 
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Element/Interface  Development 
Test  Element  -  Example 


TEST  ELEMENT  -  Unmanned  Aerial  System 


(Example) 


1.3.F1.E1.P1  %  via  direct  route. 


1.3.F1.E1.P2  %  via  waypoints. 


Development  Keys 

•  Data  Products. 

♦  Data  requirements  for  each  MOP  are 
translated  into  the  data  products. 

♦  Requirements  should  be  of  sufficient 
detail  to  provide  the  scope  of  the  effort 
that  will  generate  the  data  product. 

♦  Each  MOP  must  have  at  least  one  data 
product. 

♦  More  than  one  MOP  can  be  supported 
by  a  data  product. 

•  Data  Sources. 

♦  Data  sources  can  include:  contractor 
tests,  developmental  test,  operational 
tests,  field  exercises,  and  modeling 
and  simulations. 

•  Evaluation  Strategy. 

♦  The  test  element  describes  an 
integrated  test  program. 

♦  The  test  element  also  provides  a  method  to  view  the  acceptability  of  the  entire  evaluation  strategy. 

=  Are  the  data  products  sufficient  to  evaluate  the  MOE/MOS  standard? 

=  Which  functions/tasks  are  demonstrated  solely  in  DT? 

=  Are  there  any  functions/tasks  that  are  not  demonstrated  prior  to  OT?  Is  this  acceptable? 


Military 

Judgment 


1.3.F1.E1%of 
Successful  Course 
Changes 


X 

X 

2.C1.S1.P1  MTTR 

>100  hrs  (KPP) 

2.C1.S1  Reparability 

X 

X 

2.2.F1.E1.P1  Stationary  Targets 

>  XX.X  %  at  XX 
km  (KPP) 

2.2.F1.E1  %of 

Targets  Detected 

X 

X 

X 

2.3.F1.E1.P1  Difference 
between  estimated  and  actual 
time  of  arrival. 

<10  sec  from 
estimated  time 
of  arrival  (AA) 

2.3.F1.E1  Waypoint 
Arrival  On-Time  % 

X 

X 

X 

X 

X 

2.3.1. F1.E1.P1  %  of  Successful 
T/0 

Must  Control 
YES/NO  (AA) 

2.3.1. FI. El  %of 
Successful  T/0 

X 

X 

X 

X 

X 

2.3.1. FI. E2.P1  %  of  Successful 
Landings 

Must  Control 
YES/NO  (AA) 

2.3.1.F1.E2%of 
Successful  Landings 

X 

3.C1.S1.P1  MTBSF 

X 

>  50  hrs  (KPP) 

3.C1.S1  In-flight  Rel 

X 

3.1. FI. El. PI  %targets  hit. 

X 

Performance 
similar  to  AGM- 

XXX 

3.1.F1.E1 

Probability  of  Single 
Shot  Flit 

0.1. El. PI  Time  to  plan. 

Military 

Judgment 

0.1. El  %of 

Successful  Mission 
Planning  Sessions 

0.1.E1.P2  %  successful  loads. 

X 

0.2.3.3.E1.P1  %  operational 
targets  detected. 

>  XX.X  %  at  XX 
km  (KPP) 

0.2.3.3.E1  %  of 
Targets  Detected 
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Element/Interface  Development 
Test  &  Evaluation  Plan 

•  Documents  the  four  elements  and  the  interfaces  between  them. 

•  Two  main  body  chapters:  mission  evaluation  and  data  sources. 


MISSION  EVALUATION  CHAPTER  DATA  SOURCES  CHAPTER 


Mission 

Data  Sources 

Description  of  the  overall  mission. 

Summary  description  of  all  data  sources. 

-  Mission  Task 

Summary  data  product  schedule  for  all 

Description  of  the  mission  task. 

data  sources. 

System  functions  input  rule. 

-  Data  Source 

Conditions. 

Purpose  and  description  of  the  data 

-  Measure  of  Effectiveness 

source. 

Description  of  the  MOE. 

Scope  and  schedule  of  the  data 

Evaluation  Design  and  Procedure. 

source. 

Standard. 

-  Data  Products 

-  Measure  of  Performance 

Description  of  the  data  product. 

Description  of  the  MOP. 

Listing  of  the  MOPs  requiring  the 

Method  of  Analysis. 

-  Data  Product  (s) 

data  product. 

Listing  of  required  data  product  (s). 

-  System 

-  MOS;  MOP;  Data  Product  (s). 

-  Svstem  Function  .  ^ 

-  MOE;  MOP;  Data  Product  (s). 

r 
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Application 

Test  and  Evaluation  Elements 

•  Test  Element: 

♦  Data  is  collected  from  the  data  sources. 

♦  Data  is  then  authenticated  in  terms  of  quantity,  quality 
and  applicability. 

♦  Authentication  body  (Data  Authentication  Group) 
includes  representatives  from  the  test  events,  other  data 
sources,  the  evaluator  and  materiel  developer. 

•  Evaluation  Element: 

♦  Data  is  then  organized  and  analyzed. 

♦  Each  MOE/S  is  rated  as  met  or  not  met  based  on  the 
standard. 

♦  The  ratings  are  used  to  determine  system  capabilities 
and  limitations  and  mission  abilities  and  restrictions. 
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Application 
System  Element 

•  System  item  capabilities  and  limitations  are  determined  at  the  System  Element. 

♦  Capability:  “The  (system)  has  the  capability  to  (function  capability  with  reference  to  standard).” 

♦  Limitation:  “The  (system)  is  limited  to  (function  capability)  which  is  (shortcoming  with  reference 
to  the  standard).” 


•  MOE/MOS  ratings  are 
applied  to  the  system 
functions  to  determine  the 
system  capabilities  and 
limitations. 

•  Capabilities  and  limitations 
of  lower  level  system 
functions  are  also  used  to 
evaluate  higher  system 
functions. 


•  Tool  developed  to  resolve 
the  system  functions. 
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Application 
Mission  Element 


•  Mission  task  capabilities  and  limitations  are  determined  at  the  Mission  Element. 


♦  Capability:  “The  (unit)  has  the  capability  to  (task  ability)  while  (task).” 

♦  Limitation:  “The  (unit)  is  limited  to  (task  ability)  while  (task)  which  is  (shortcoming  to  mission 


Application 
Evaluation  Report 

•  The  evaluation  report  provides  the  documented  results  of  the  evaluation. 

♦  Mission  Evaluation  Results. 

«  Mission  performance  in  terms  of  mission  threads, 
e  Overall  mission  capabilities  and  limitations. 

«  Individual  mission  task  capabilities  and  limitations. 

♦  System  Evaluation  Results. 

«  System  performance  in  terms  of  attributes  and  KPPs. 
e  System  effectiveness  and  suitability, 
e  Overall  system  capabilities  and  limitations. 


Provides  the  decision  maker  with. . . 

a  clear  picture  of  the  system  capabilities  and  limitations  allowing  acquisition  decisions 
^^ased  on  the  military  utility  gained. 


Provides  the  warfighter  with. . . 

a  clear  picture  of  the  unit’s  capabilities  and  limitations  within  the  context  of  the  mission. 
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Strengths 

•  Provides  a  mission-based  form  of  evaluation. 

♦  Military  utility  of  the  system  immediately  apparent  to  the  user. 

♦  System  suitability  directly  linked  to  mission  capability. 

•  Outlines  a  fully  integrated  test  and  evaluation  program. 

♦  Promotes  synergistic  use  of  data  gathered  from  all  sources:  contractor  test, 
developmental  test,  operational  test,  and  modeling  and  simulation. 

♦  Promotes  early  identification  of  T&E  strategy  risks. 

•  Provides  continuous  evaluation  of  the  mission  throughout  all 
system  development  phases. 

♦  Impact  of  development  risks  on  the  mission  visible  in  early  development. 

♦  Monitors  progress  of  system  development  and  demonstration  within  the 
context  of  mission  capabilities  provided. 

♦  Incremental  development  strategies  are  supported  by  evaluating  each 
increment’s  capabilities  in  the  context  of  the  overall  mission. 
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Concept  Development 

Path  Forward 

•  Joint  Cargo  Aircraft  Case  Study:  Program  Currently  in  T&E  Planning. 

♦  GOAL:  Produce  an  executable  System  Evaluation  Plan. 

♦  GOAL:  Further  develop  the  end-product  (completed  mission-based  evaluation). 

•  Case  Study:  TBD  Program  Currently  in  T&E  Concept  Stage. 

♦  GOAL:  Demonstrate  a  Fully  Integrated  T&E  Concept  from  MS  B  to  Full-rate  Production  Decision. 

•  Tool  Development:  Develop  tools  to  support  development  of  elements  and  interfaces  and  execution 
of  the  T&E. 

CH-47F  Case  Study  -  Results  &  Lessons  Learned 

•  All  four  elements  and  the  interfaces  successfully  developed  in  a  spreadsheet. 

•  Verified  that  the  concept  can  be  applied  to  build  a  mission-based  T&E  strategy. 

•  Some  system  functions/characteristics  may  require  more  than  one  sub-system  to  execute. 

♦  Define  an  “integrated  system/sub-system”  level  in  system  element  to  support  these  functions/eharaeteristies. 

•  Information  Exchange  Requirements/Net  Ready  not  defined  at  the  system  level. 

♦  Two  options: 

=  Define  whieh  system  or  sub-system  provides  the  interoperability  eapability  and  link  the  evaluation  requirement  to  that 
system’s  funetion  or  eharaeteristie; 

=  or  link  the  evaluation  requirement  to  the  function  or  characteristic  of  an  integrated  system. 

•  Need  more  applicable  standards. 

♦  By  Observation:  No  defined  standard.  Capabilities  and  limitations  reported  as  observed. 

♦  Descriptive:  No  defined  standard.  Data  taken  to  support  determination  of  capabilities/limitations. 

♦  Defined  Standard:  Standard  given  lAW  military  specification,  regulation,  etc. 
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Summary 

•  Mission-based  evaluation  proeess  has  been  developed  to  support 
T&E  planning  and  execution.  Process  is  comprised  of: 

♦  Four  elements. 

«  Mission  Element - Mission  Tasks  and  Sub-tasks. 

s  System  Element - System  Items  and  Functions. 

s  Evaluation  Element  — Evaluation  MOEs  and  MOPs. 

«  Test  Element - Data  Sources  and  Products. 

♦  Interfaces. 

«  Links  between  each  element  have  been  developed  to  facilitate  T&E  planning  and 
execution. 

•  Execution  of  the  T&E  effort  provides: 

♦  the  decision  maker  with  a  clear  picture  of  the  system  capabilities  and 
limitations  allowing  acquisition  decisions  based  on  the  military  utility  gained. 

♦  the  warfighter  with  a  clear  picture  of  the  unit’s  capabilities  and  limitations 

within  the  context  of  the  mission.  25 
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Element,  Links  &  Traces 


ELEMENTS  AND  TRACES 


r  ELEMENT 

h 


PLANNING  TRACE 
EVALUATION  TRACE 

ELEMENT  TO  ELEMENT 
INTERFACE 


SYSTEM 

ELEMENT 

— ^ - 


EVALUATION 

ELEMENT 


MISSION 

ELEMENT 

f  f  f 
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Acronym  Chart 


AA 

Additional  Attribute 

MOS 

Measure  of  Suitability 

AV 

All  View  (slide  4) 

OA 

Operational  Area 

AV 

Air  Vehicle  (slides  11,  13,  andl5) 

OT 

Operational  Test 

CDD 

Capabilities  Development  Document 

OT&E 

Operational  Test  and  Evaluation 

COI 

Critical  Operational  Issue 

OV 

Operational  View 

CPD 

Capabilities  Production  Document 

RSTA 

Reconnaissance,  Surveillance  &  Target  Acquisition 

DAG 

Data  Authentication  Group 

RT 

Remote  Terminal 

DoD 

Department  of  Defense 

SATCOM 

Satellite  Communications 

DT 

Developmental  Test 

sv 

Systems  View 

GCS 

Ground  Control  Station 

T&E 

Test  and  Evaluation 

JCIDS 

Joint  Capabilities  Integration  and  Development  System 

T/0 

Takeoff 

KPP 

Key  Performance  Parameter 

TM 

Telemetry 

MER 

Mission  Evaluation  Report 

TV 

Technical  View 

METT-TC 

Mission,  Enemy,  Terrain,  Troops,  Time  and  Civil 

UAS 

Unmanned  Aerial  System 

MOE 

Measure  of  Effectiveness 

UAV 

Unmanned  Aerial  Vehicle 

MOP 

Measure  of  Performance 
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